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ABSTRACT 

This paper outlines the design 
. udies of Phase IV of the Individually Guided 
Evaluation Project. More than 2,000 elementary 
use the IGE system. The Evaluation Project was 

comprehensive view of the system's oper»ition and effectiveness. Phase 
IV investigated pupil outcomes, instructional time, and" means of 
instruction. It focused on two IGE curriculum programs designed for 
compatibility with instructional programming for the individual 
student: (1) the Wisconsin Design for Reading Skills Development 
(WDRSD) ; and (2) Developing Mathematical Processes (DMP) . Second and 
fifth grade students and their teachers in three types of schools 
were studied: (1) IGE schools using DMP or WDRSD; (2) Non-IGE schools 
using DMP or WDRSD; and (3) IGE schools using neither program. Data 
were collected during the 1978-79 school year from achievement 
monitoring and domain referenced tests, teacher logs, clabsroom 
observations, and questionnaires which served as the basis for 
structured interviews wi'.h school staff. Pupil attainment of 
specified program objectives was contrasted in IGE schools using 
different programs and in IGE and non-IGE schools using WDRSD or DMP. 
Regression analyses were performed using the residual gain score as 
the dependant variable and instructional time, means of instruction, 
and school variables as the independent variables. (BS) 
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MISSION STATEMENT 



Ihe mission of the Wisconsin Research and Development Center 
IS to improve the quality of education by addressinq the full 
range of issues and problems related to individualized schooling. 
Teaching, learning, and the problems of individualization are 
given concurrent attention in the Center's efforts to discover 
processes and develop strategies and materials for use in the 
schools. The Center pursues its mission by 

• conducting and synthesizing research to clarify the 
processes of school-age children's learning and 
development 

• conducting and synthesizinq r»^^;earch to clarify effective 
approaches to teaching students basic skillri and concepts 

• developing and demonstrating improved instructional strategies, 
processes, and materials for students, teachers, and school 
administrators 

• providing assistance to educators which helps transfer the 
outcomes of research and development to improved pi'actice 
in local schools and teacher education institutions 

Tiie Wisconsin Research and nevplo[jment Cent.^r i.^; :>uf:rn)rted 
with funds from the National Institute of Education and the 
University of Wisconsin. 
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This paper outlines the design of two Comparative Studies of 
Phase IV of the Individually Guided Education (IGE) Evaluation Pro- 
ject, Phase IV is one of five related phases comprising an extensive 
study designed to evaluate IGE. It attempted to describe how instruc- 
tion in reading skills and mathematics is carried out in IGE and non- 
IGE settings in which curricular materials designed to be compatible 
with the IGE instructional programming model were used. 

IGE and the Evaluation Project 

Through the combined efforts of the Wisconsin Research and Develop- 
ment Center for Individualized Schooling, the. University of Wisconsin 
IGE Teacher Education Project, the Kettering Foundation (I/D/E/A), and 
IGE coordinators in 25 states, more than 2,000 elementary schools have 
adopted a system called Individually Guided Education (IGE). 

IGE is a complex system based on theoretic and pragmatic ideas 
about schooling, children's learning, and the professional roles of 
school staffs. This system contains seven components: 

1. Multiunit organization, 

2. Instructional programming for the individual student, 

3. Assessment and evaluation for educational decision making, 

4. Curriculum and instructional materials and activities for 
each child's instructional program, 

5. Home-school-community relations programs, 

6. Facilitative environments for professional growth, and 
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7. Continuing research and development for system improvement. 
Each of these seven components was developed as the result of an 
extensive collaborative study by scholars and professional educa- 
tors . 

Four types of variables were identified to guide the evaluation 
of IGE: pupil and staff outcomes, means of instruction, support 
systems, and pupil and staff background. With these variables in mind, 
a descriptive framework was developed that considers outcomes of IGE as 
a function of both instructional means and the degree of implementation 
(Romberg, 1976). Figure 1 shows how the four types of variables are 
related . 

1 . Pupil and staff outcomes, and the extent to which these outco mes 
have been attained, should be the initial basis of an IG E evaluation . 
Both pupil and staff outcomes are illustrated in Figure 1 ?s being 
multivariate and multilevel. In this study a set of curriculum-specific 
pupil achievement scores in reading and mathematics wil] be used. 

2 . The instructional means of formal schooling must be a secon d 
basis fr^r an evaluation of in E> It has been fashionable in evaluation 
circles to concentrate on ends or outcomes and to ignore the means by 
which they are reached. Refortn movements, such as IGE, invariably 
attack the properties of means. To this extent, judging the value of 
Dhe means is as important ac assessing outcomes. 

The means uf ins true Lion considered in the evaluation project were 
separated into three sets of activities based upon the operating 
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characteristics of IGK sdhools: staff activities of the Instructional 
Improvement Conunittee (IIC) and the Instruction and Research Units 
(I & R Units), activities of the staff teacher both in curriculum 
management and pupil interactions, and activities of pupils as they 
are related to reading and mathematics instruction. It is this latter 
set of activities that is to be' examined in detail in the Comparative 
Study of Phase IV. 

3. The degree to which^ and the way in which, support systems of 
IGE have been incorporated and developed in a school must be judge d. 
The seven components of IGE have evolved as practical features which are 
needed to support new instructional methods which in turn produce 
desired pupil and staff outcomes. It can be argued that the efficiency 
of an IGE school depends upon the components which have been implemented 
and the manner in which they are operating. 

The support systems for an IGE learning environment were separated 
into four categories as indicated in Figure 1. The second category, 
curricular materials compatible with instructional programming and 
evaluation (IGE Component ^0, is shown by identifying^ the three maior 
curricular products developed for IGE, the Wisconsin Design For Reading 
Skill Development (WDRSD ) (Otto, 1977), Developing Mathematical 
Processes (DMP ) (Romberg, 1977), and the Pre-Readtng Skills Program 
(PRS) (Venezky & Pittelman, 1977). The functional relationships 
illustrated in Figure 1 convey the following premises: (a) the degree 
to which IGE support systems have been implemented, together with 
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pupil and staff backgrounds, directly influences the means of 
instruction in an IGE school; and (b) the means of instruction, along 
with pupil and staff backgrounds, account for pupil and staff butcorr.es. 

Although miu'h has been written about: t\w conceptual backj>round of 
TOP, no comprehensive picture now exists which shows the manner in 
which IGE has been implemented in elementary schools. Thus, the ICE 
Evaluation Project was designed to evaluate IGE in order to gain a more 
comprehensive view of the system^s operation and effectiveness. The 
desired outcome is to identify which features contribute most to the 
success of reading skills and mathematics instruction as a result of 
individualized schooling. 



The evaluation project consisted of five phases which were 
organized to provide complementary information of individually guided 
instruction. Phase T was a large sample study which provided basic 



have been reputed to be crucial to IGE success. The purpose of 
Phase T, then, was to examine the extent to which those presumably 
i^sqential features had been implemented among IGE schools and to 
assess the effectiveness of that implementation. In this large sample 
study, including approximately 155 IGE schools, information was 
obtained from IGE school staff members using self-report surveys and 
from students using standard paper-and-pencil instruments. The data ' 
were intented to provide a functipnal understanding of IGE features, 



information about IGE schooling, v^ertain features of IGE schooling 
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processes, and outcomes by relating a broad scope of variables in 
an interpretive manner (Price, Janicki, & Romberg, 1980). 

Phase II verified and extended the self-report data gathered in 
Phase I to include more fully the range of variables that determine 
the process of schooling (Ironside & Conaway, 1979). 

•Phase III focused on the social meaning which emerges as IGE is 
implemented on a day-to-day basis. The problem of understanding the 
impact of educational reform can be approached by viewing schools as 
social institutions whose characteristics shape and are shaped by the 
behaviors of their members. This focus allows us to think of a school 
as a complex social arrangement consisting of underlying patterns of 
conduct which channel thought and action within that setting (Popkewltz, 
Tabachnick, & Wehlage, in press). 

Since the success of IGE depends heavilv on the availability of 
naterials and evaluative procedures which are compatible with 
hisr ru^'t ional programming for the individual student, an analvsis or' 
cuirriculum products designed to be used in T'OF. settings was undertaken. 
This aspect of the Phase IV project sought to determine the extent to 
which the three curricular programs developed for ICR met their 
objertives and tr clarify the relationship of Pupil outcomes to 
instructional timt and means of instruction. In addition, Phase IV 
provides information about pupil activities and learning outcomes 
as they relate to specific objectives. 
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Finally, the p,oal of Phase V Is to synthesize the results qf 
Phases I through IV and to address the significant issues in 
contemporary schooling raised by the project as a whole. Thus, each 
phase of the evaluation Was designed to complement and strengthen the 
validity of the data gathered by the previous phases. For example, 
data on means of instruction, gathered by the large-sample study of 
Phase f ^^^Ce examined in somewhat greater depth in f^wer schools by 
the Phase II studies. Phase III^s analysis develops a view of 
Instruction from a different perspective. Phase IV explores means a£ 
instruction within the specific curricular areas of reading skills and 
mathematics. Thus » instead of merely adding together summaries of the 
different evaluation phases, Phase V is designed to integrate and 
interpret the data from all the phases into a series of statements 
about what implications the project has for educational Iss-ues. 

Overview of Phase IV 

The intent of Phase IV was to describe in conis iderab le 
dot:n;i the nctual operating characteristics of n sample of schools 
which wertj ysing curriculum materials designed to be compatible with 
IGE. Phase IV was restricted to the investigation of thrp»e grotips of 
variables — pupil outcomes, instructional Lime, and means of 
Inst niction-in IGE and non-IGE settings in which the Center's 
curriculum programs as well as alternative curriculum materials were 
being used. Pupil attainment of program objectives is the main 
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variable. The other two variables, instructional time and means of 
instruction, are essential in explaining and understanding how the 
programs work and how objectives are obtained. These two variabl"&?5^' 
are also impotLant from a practical point of view because they are 
variables that can be manipulated by teachers. Describing the use 
of each program in terms of allocated time, engaged time, and 
instructional activities provides concrete factors that teachers ♦ 
work with, 

Tn addition, instructional time was included because of recent 
studies and reviews that stress its importance and its relationship 
to pupil outcomes (Harnischf eger & Wiley, 1975; Rosenshine, 1977; 
McDonald & EliavS, 1976; and Fisher et al.,, 197 5). As Harnischf eger 
and Wiley ^tate, **A11 influences on pupil achievement must be mediated 
through a pupil^s active and passive pursuit** (1975, p. 15) • Certainly, 
there is enough evidence to suggest that instructional time, as a 
' measiire of pupil pursuits, is Important. Its use as a variable in 
Phase TV, then, has two purposes. Firat, the amount of time during 



wliich stiidents are actively engaged in learning when using one of the 
tiiree programs will be a means of describing how the programs are 
buing used. The assumption is that the programs should maximize 
student engagement by attending to the individual's needs. Second, 
Phase IV provides an excellent opportunity to study in more detail 
the relationship of pupil outcomes to instructional time. 
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In summary, the primary purposes of Phase TV are: 

1. to determine the degree to which WDRSD and DMP meet their 
objectives of having student3 master specified objectives 
and skills; 

2. to determine how time is allocated for instruction in 
implementing WDRSD and DMP; 

3. to relate instructi-onal time to the means of instruction and 
mastery of content for WDRSD and DMP; and 

4. for each curriculum program, WDRSD and Dl'tP, to contrast 
two situations-IGE schools using the program with non-TGE 
schools using the program and TGE schools using the program 
with ICE schools using alternative programs — on the variables 
of pupil outcomes, instructional time and means of instruc- 
tion. 

Five studies were conducted as part of Phase TV, three Descriptive 
Studies and two Comparative Studies. The Descriptive Studies were 
small sample studies designed to describe how the curriculum programs 
DMP, IVDRSD, and PRS were being used in TGE schools. The studies were 
conducted during the winter and spring of 1978 at two TGE schools using 
DMP, two TGF schools using WDRSD, and three TGE schools using PRS. Achieve 
ment monitoring and domain referenced tests, observations, teacher logs, 
and questionnaires were used to collect the data, and one purpose of the 
Descriptive Studies was to refine these procedures for subsequent use 
in the Comparative Studies. A more detailed description of the Descrip- 
tive Studies is given in Project Paper 79-42 (Webb 6i Romberg, 1979). 
'Yhkj two Comparative Studies also focused on the use of WDRSD and DMP 
in IGE settings. 
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The Comparative Studies 
Data were gathered for the Comparative Studies from October 

until May during the 1978-79 school year. Three types of schools were 

> 

included in the study: 

1. IGE schools using DMP or WDRSD ^ 

2. Non-IGE schools using DMP or WDRSD 

3. IGE schools using neither program 

Briefly, the procedures followed In the Comparative Studies were as 
follows: Four triads of schools were selected for WDRSD and three 
triads for DMP; each triad contained one school of each of the three 
types. Only students In Grades 2 and 5 and their teachers participated 
in the study. Data were collected by four means: tests on general 
objectives of each currlcufum program, observations of specific students 
during the reading or mathematics Instructional period, teacher logs 
for reading or mathematics Instruction of specific students, and ques- 
tionnaires which served as the basis for .structured Interviews with 
school staff. The tests were administered eight times during the year 
with the first testing In October 1978 and the last in May 1979. Each 
testing was separated by approximately three weeks of Instruction, during 
which formal observations were made at each grade level with a total 
number of observations for the 25-week stody ranging from 16 to 20. In 
addition, teachers maintained dally logs for a sample of pupils over the 
entire testing observation period, recording the amount oi: time allocated 
for Instruction by objective for each target pupil as well as the type 
of materials and grouping used during Instruction. In addition, data 
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on school and staff background variables and on curriculum, use variables 

were obtained in structural interviews with teachers, unit leaders, aod 

principals. 

Model for the Studies 

A structural model for predicting student achievement was devel- 
oped for Phase I and is shown in Figure 2 (Price, Janicki, Howard, 
Stewart, Buchanan, & Romberg, 1978). This model was developed from 
the three premises on which IGE is based. They are; 

1. Certain organizational features make it more likely that ; 
certain desirable instructional practices will occur. 
These organizational features also make it more likely 
that the staff will be satisfied with their jobs. 



2. %The use of certain curriculum materials and associated 

systems of information collection and record keeping 
makes it more likely that certain desirable Instructional 
practices will occur. 

3. Those instructional practices which are deemed desirable 
in TGE make high student achievement more likely. They 
also make it more likely that desirable changes in other 
student characteristics, such as self -perception and 
locus of control, will occur. 

Data on the first . premise were organized into six variables: Intc»r- 

orj^^anizat ional Relations /lOR). Procedures Postering Coordination nnd 

Improvement of the School Program (COS), Intraorganizat lonal Structure 

(lOS), General Staff Background (GSB), Belief in Individual Differences 

fINDIV), and General Implementation of the Instructional Programralnj^ 

Model (IPM). These describe in detail the organizational structure 

and staff background in the school. Four variables were developed in 

response to the second premise. These curriculum-specific variables nro 

Utilization of Curriculum Resources (UCR) , Information Acquisition (lA) , 

Individualization of Instructional Decisions (IT)M), and Management of 
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Grouping and of Instructional Continuity (IE), A single variable- 
Student Achievement — includes tUe Rupil outcomes^^scussed in the third 
premise* With the exception of the achievement measures > data on all 



general staff, and principal questionnaires • 

In contrast, Phase- IV was designed to provide more detail on the 
last two premises posed in Phase I, with specific attention paid to 
means of instruction and curriculum^related student achievement, while 
providing sufficient background information that each school in the 
smaller Phase IV sample might be related on several significant di- 
mensions to schools in the larger Phase I sample as a whole* Thus, 
some information was collected on five of the six school-wide vari- 
ables used in Phase I— General Implementation of the Instructional 
Programming Model (IPM), Intraorganizational Structure (lOS), Proce- 
dures Fostering Coordination and Improvement of the School Program (COS), 
Interorganizational Relations (lOR) and General Staff Background (GSB) • 
In Phase IV the Program Use variables—Curriculum Implementation and 
Program Customizing — included the kinds of information provided in the 
Phase I currlculum^specif ic variables* More detailed information about 
classroom procedures and achievement outcomes was also collected in 
Phase IV. A model depicting the Phase IV variables and the anticipated 
relationships is shown in Figure 3. 

Four groups of variables are shown in Figure 3 — school background, 
curriculum program uss, classroom activities, and pupil outcomes. As 
stated above, the school variables, which were assessed through structured 



of the variables in the Phase I model were collected using teacher. 
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interviews with school staff, provide a link between the Phase IV 
sample and the larger Phase I sample. Curriculum program use variables, 
also measured through structured interviews, have a linking function to 
Phase I and provide a descriptive background for the measures of class- 
room procedures. These procedures were assessed through logs maintained 
,by teachers for selected students and through observations in the class- 
rooms; means of instruction and the use of instructional time are de- 
tailed measures of how programs are used in classrooms and relate di- 
rectly to pupil attainment of objectives. Pupil outcomes have been 
specified in terms of specified objectives of the curriculum programs; 
they are assessed through achievement monitoring procedures and, for 
the DM? Study, domain referenced procedures. 
Sample 

Four WDRSD triads and three DMP triads were identified to partici- 
pate in the studies. Each triad was to have one school of each of the 
three types. Schools within each triad were matched according to loca- 
tion, socio-economic level, composition of student body, size, and, for 
the IGE schools, "IGE~ness." The same demographic categories used in 
Phase I were used to classify the communities in which the schools were 
located: 

1. Extreme rural— community under 3,500, residents primarily farmers 
or farm workers 

2. Small place — community under 25,000 

3. Medium city — ^community between 25,000 and 200,000 
A. Main big city— community within a city over 200,000 
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5. High raetro — area in a city with a population over 150,000 
where many residents are in professional or managerial po*sitions 

6. Low metro— area in a city with a population over 150,000 where 
many residents are on welfare or are not regularly employed 

7. Urban fringe— suburb of a city greater than 200,000 

The four triads of schools in the WDRSD study represented extreme rural, 
small place, medium city, and urban fringe. One urban fringe IGE school 
not using WDRSD withdrew from the study just prior to the beginning of 
the data collection. Thus, the urban fringe group was reduced to two 
schools, an IGE school and a nofi^IGE school, both using ITORSD, bringing 
the total number of schools in the WDRSD study to eleven* The three 
triads of schools in the DMP study represented extreme rural/small place, 
medium city, and urban fringe. One IGE school using DMP from a medium 
city withdrew from the study at the last minute reducing the number of 
schools in the DMP study to eight. 
Data Collection 

Pupil outcomes .. An achievement monitoring procedure with item sampl- 
ing was used to collect data on the attainment of objectives. Both the 
DMP and WDRSD programs contain units of instruction — topic for DMP and 
skills for WDRSD — and are based on the instructional programming model 
(IPM). Once a pupil has mastered the objectives of a topic or skill, 
the pupil is to be regrouped with other pupils with similar needs and 
given instruction on a new topic or skill. The instructional sequence 
of topics or skills should vary from pupil to pupil. Because of this 
variation in the instructional programs which pupils receive, an achievement 
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monitoring procedure in which tests were administered at eight points 
during the school year was chosen to provide information on the attain- 
ment of objectives » Such a procedure is more sensitive to the individu- 
alization of the programs than other designs, such as pre- and posttest- 
ing. 

The tests' used in the Descriptive Studies (Webb & Romberg, 1979) 

1 

were refined for use in the Comparative Studies. The tests were compiled 
by identifying 25 WDRSD Skills for Grade 2, 26 WDRSD Skills for Grade 5, 
12 DMP objectives for Grade 2 and 14 DMP objectives for Grade 5. Two 
to four test items for each of the WDRSD Skills and four test items for 
each of the DMP objectives were then prepared to form an item pool for 
each grade level for each program. Items from each pool were distributed 
among four forms using an item sampling technique. For the Comparative, 
Studies, all achievement monitoring test items were constructed in a 
multiple-choice format and used terminology which would be understood 
by pupils using programs other than the curriculum under consideration. 

The nchlevement monitoring tests were administered eight times during 
the school year. The pupils at each grade level were divided at random 
into four groups and the four test forms at each level were rotated nmon}^ 
the groups so that each group was given a different form of the test for 
any consecutive testing and^ over the school year, each student .took each 
form twice. The maximum time for any one testing for a student was 50 
minutes. 

A second testing procedure was used to measure achievement on three 
specific math objectives at each of the two grade levels at the eight 
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schools participating in the mathematics study. Domain-referenced 
tests were administered three times — at Test Time 1 (Octotjer, at Test 
Time 4 (January), and at Test Time 8 (May). This procedure was used 
«. , in order to test key objectives in more detail and to provide a measure 

of pupil outcomes on the general concepts and ideas associated with 
the objective domain as well as specific skills needed to perform the 
individual items. For example, for the Grade 2 objective of counting 
• objects ip sets from 0-99, the analysis of the domain-referenced tests 
provided measures of the ability of the group of pupils to count the 
objects as well as the pupils* specific problems in counting- obj ects 
grouped in different ways (e.g., by fives, sixes, and tens). The domain- 
referenced procedure was not used in the reriding skills Comparative Study 
because of the results of using this procedure in the WDRSD Descriptive 
Study. Identifying the domains for the specific reading skills was much 
more difficult than for math objectives and resulted, in general, in 
spurious results. 

For each of the three test times the domain-referenced tests were 
administered, a set of ten items was randomly chosen from the specified 
dpmain of item& for each of the three objectives for the grade level. 
All^items were open ended and required the pupils to supply the answer 
in order to minimize guessing. The Harris-Pearlman (1978) procedure 
was used to separate the item difficulty into two factors, one repre- 
senting the domain difficulty and the othe*" representing difficulty with 
the specific item* Also, since all students took the same tests, the 
domain-referenced tests can be used to compare the representativeness 
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of the outcomes of the tatget students to the group from which they 
were selected. The testing time for the domain-referenced tests 
was approximately AO minutes. 

Observations . The observation system is the same as the one Lri 
the Phase IV Descriptive Study (Webb & Romberg, 1979). Initially, 
six target students were randomly identified in the unit or class. 
The target students changed over the year, since in some IGE situations 
students are regrouped periodically, making it often physically 
impossible to observe the same six students. These students were ob- 
served in sequence using a time sampling procedure. The first target 
student was observed for a moment and his or her activity was coded. 
Then the next tar^oX student was observed for a moment and his or her 
activity coded. The procedure continued until all six target 
students had been observed, taking approximately 3 minutes., rhirtv 
seconds were then taken to record the major role of the teacher(s) 
and general activities occurring in the classroom. This cycle was 
re[)eated , observing each target student in sequence ^^nd record i n^ 
v^eneral cf^nimcnts, during the time allocated for work on the curriculum 
program. 

Se^ven major iMtegories of data were coded: 

1. Heneral content — time devoted to other than the curricular 
program being observed 

J, Specific content — math objective or reading skill 

3. Pace — wliether or not the student is worklnj; at his ur [u'r 
own pace 

4. Groui^in^ — size of group of whicli the student is a member 
"i. Materials — the materials being used bv the student 
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6. Learner moves—student engagement or non-engagement 

7. Interaction — persons with whom the student is interacting 
and the direction and focus of that interaction 

The event occurring at the moment the target student was observed 

was characterized by checking subcategories under each of these main 

categories. This observation . system was us§d to provide measures on 

the amount of time spent in general content areas such as waiting, 

transition, and management and, for specific content areas in reading 

and math, measures of t-he amount of time spent by students with 

different ^.ypes of groupings, materials, and interactions as well as 

1 

' different types of engagement. 

The obs^rv^s were trained to use the observation svstem in a 
four-day training workshop held in Madison in October 1978. Th^ 
first day of the workshop was spent reviewing the materials and 
procedures 'used in each of the programs and explaining the observa- 
• tlon system. Then the observers spent three days at a school doing 
observations and discussing the coding procedures. Percentage 
agreement on individual events and intercoder rellabilit-ies on sums 
over events were calculated to assess the level of proficien))/ that 
the observers had attained in using the observation procedures. In 
addition, a sample of schools were visited during the year to check 
the percentage agreement and intercoder reliability. The observer"? 
also returned for a twn-day retraining session in February 1979, most 
of which involved observations in schools to check on the intercoder 
reliabilities.' 
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Teacher logs . Teachers who were directly responsible for the mathe- 
matics or reading skills instruction of the students in the target popu- 
lation were asked to keep daily logs for six to eight students', including 
those students being observed, in order to obtain a measure of the total 
time allocated to instruction on specific objectives over the investiga- 
tion period • On the logs the teachers recorded the amount of instruc- 
tional time which was allocated to specific math or reading skills as 
well as the group size and type of materials used during instruction. 

Interviews . In each school, structured inteirviews were conducted by 
the observer for that school with members of the Grade 2 and Grade 5 in- 
structional staff and with the principal. Background information about 
the school, the staff, and use of the reading or mathematics curriculum 
products was obtained from these interviews. The questionnaires used 
as the basis for the interviews were developed from two sources:, the 
Phase I survey instruments and the curriculum developers' questionnaires 
about product use. 

Instructional staff provided information about their own teaching 
experience, how the curriculum product was used, and how the* overall 
instructional program was planned and carried out. Each principal 
described the school's organization, its relationship toother educa- 
tional agencies, and some procedural aspects of the school's ongoing 
operation . 

Data analyses '. The analyses of the data for the Comparative Studies 
were guided by the four purposes related to the general goals of Phase 
IV (see page 8) . 

/ 



25 



19 



Central to the analyses are two contrasts': IGF schools using VTORSD 
or DMP with IGE schools using alnernative programs, and IGE schools using 
WDRSD or DMP with non-IGE schools using, WDRSD or DMP, Pupil attainment 
of specified program objectives thus was contrasted in IGE schools using 
different programs and in schools that were IGE and non-IGE using the pro- 
gram under study, WDRSD or DMP, Additional contrasts were made related to 
allocated time, pupil use of instructional time, and the means of instruc- 
tion provided. Analyses were performed by grade level. Grades 2 and 5, 
for each general objective area within reading skills and tnathematics, the 
unit of analysis being the school. In addition, the objectives for mathe- 
matics and reading skills were aggregated for analysis to form more general 
content areas and some aggregations were done on the means of instruction 
variables by consolidating grouping, materials, and some of the interaction 
categories. 

In describing the variables for the individual schools and school/ 
program types, the objective easiness achievement measures^ (a percentage 
of correct responses to total possible responses) were reported for each 
content area. Residual gain scores (final achievement less the predicted 
achievement based on prior achievement scores) were used as measures of gain 
in achievement. Time was used as the metric for describing the means of 
instruction variables: The time observed for each variable was used to 
determine the percentage and relative use of the variable. 

A measure of the content covered was provided for each general ob- 
jective by reporting the total time in hours allocated to the objective 



and its percentage of the total time allocated to raatliematlcs or readipg 



1 
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skills* Allocated time was determined by teacher logs. Of interest are 
the extent to which the content covered varied by school type and cur- 
riculum program or by school within a type as well as ihe extent to 
which there are similarities among schools using the same program* 
School and background information were reported from the information 
col^cted by the questionnaires. 

For the math schools only, a more refined look at achievement on 
three specific objectives for each grade was done using the results 
of the three administrations of the domain-referenced tests* The 
Harris-Pearlraan (19/9) analysis was used to identify the proportions 

A 

of the item difficulty due to the domain (k) and due to the individual 
item. These statistics were used to report the change in achievement 
on the general concept of the domain as well as on specific skills 
within the domain. 

The relationships among variables will be determined by regression 
analyses. Regression analyses will be performed using residual gain 
score as the dependent variable and instructional time, i;iieans of instruc- 
tion, and school variables as rhe independent variables. The purpose 
of tlie regression analyses is to determine which variables have the 
^',reatertt value in predicting achievement gain. The number of variables 
that uan be included in a regression equation is limited, however, be- 
CcKise of the relative small number of schools in the studies — eleven for 
reading skills and eight for mathematics. 

After examining the base data, a set of from 6 to 10 priority questions 
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were identified and examined in detail via iterative analysis. The 
answers to these questions will be sutnmarized in two technical reports, 
one for the DMP study and one for the WDRSD study. Both reports will 
carefully examine the general notion that increased instructional time 
leads to increased achievement. An initial review of the data suggested 
that this notion is not always true. The reports also consider the vali- 
dity of the labels— IGE school, D>f? user, WDRSD user-- used to differentiate 
schools in the studies. 

Differences in the programs under study will result in different 
emphases in the two reports. DMP is a complete instructional program 
for mathematics. Analysis of DMP results will provide* an example of 
how any curriculum program can be analyzed and the parts of the program 
common across schools can be !• m :ied. WDRSD is a management system 
for reading skills instruction in general. The greater emphasis on word 
attack skills at Grade 2 and comprehension and study skills at Grade 5 
will result in different discussions for the tx>o grades. 
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